98
Binary Neural Architecture Search
Algorithm 8 ABanditNAS with adversarial training
Input: Training data, validation data, searching hyper-graph, adversarial perturbation δ, adver-
sarial manipulation budget ϵ, K = 9, hyper-parameters α, λ = 0.7, T = 3. Output: The remaining
optimal structure,
1: t = 0; c = 0
2: Get initial performance m(i,j)
k,0
3: while (K > 1) do
4:
c ←c + 1
5:
t ←t + 1
6:
Calculate sL(o(i,j)
k
) using Eq. 4.9
7:
Calculate p(o(i,j)
k
) using Eq. 4.10
8:
Select an architecture by sampling one operation based on p(o(i,j)
k
) from Ω(i,j) for every edge
# Train the selected architecture adversarially:
9:
for e = 1, ..., E do
10:
δ = Uniform(−ϵ, ϵ)
11:
δ ←δ + α· sign
∇xl
f(xe + δ), ye
12:
δ = max
min(δ, ϵ), −ϵ
13:
θ ←θ −∇θl
fθ(xe + δ), ye
14:
end for
15:
Get the accuracy a on the validation data Update the performance m(i,j)
k,t
using Eq. 4.11
16:
if c = K ∗T then
17:
Calculate sU(o(i,j)
k
) using Eq. 4.12
18:
Update the search space {Ω(i,j)} using Eq. 4.13
19:
c = 0
20:
K ←K −1
21:
end if
22: end while
to find the best λ. We train the structures in the same setting. From Fig. 4.3, we can see
that when λ = 0.7, ABanditNAS is most robust.
Effect on the search space. We test the performance of ABanditNAS with different
search spaces. In this part, we adopt the same experimental setting as the general NAS. The
search space of the general NAS has 7 operations. We incrementally add the Gabor filter,
denoising block, 1×1 dilated convolution with rate 2 and 7×7 dilated convolution with rate
2, until the number of operations in the search space reaches 11. In Table 4.1, # Search Space
represents the number of operations in the search space. Although the difficulty of searching
increases with increasing search space, ABanditNAS can effectively select the appropriate
operations. Each additional operation has little effect on search efficiency, demonstrating
the efficiency of our search method. When the number of operations in the search space is 9,
the classification accuracy of the model searched by ABanditNAS exceeds all the methods
with the same level of search cost.
4.3
CP-NAS: Child-Parent Neural Architecture Search for 1-bit
CNNs
Comparatively speaking, 1-bit CNNs based on handcrafted architectures have been ex-
tensively researched. Binarized filters have been used in conventional CNNs to compress
deep models [199, 99, 159], showing up to 58 times speedup and 32 times memory